Emmanuel Kasper: Using Debian and RHEL troubleshootings containers on Kubernetes & OpenShift
oc run troubleshooting-pod --stdin --tty --rm --image=registry.access.redhat.com/rhel7/rhel-tools
cvlc v4l2:///dev/video0and there you go.
awk
to print the nth column of a file: $ awk ' print $1 ' /etc/hosts
will print all IP addresses from /etc/hosts
But you can also do filtering before printing the chosen column: $ awk '$5 >= 2 print $2 ' /path/to/file
will print the second column of all lines, where the 5th column is greater than 2. That would have been hard with grep. Now I can use that to find out all deployments on my openshift cluster, where the number of current replicas is greater than 2. $ oc get deployments --all-namespaces awk '$5 >= 2 print $2 '
NAME
oauth-openshift
console
downloads
router-default
etcd-quorum-guard
prometheus-adapter
thanos-querier
packageserver
I know that openshift/kubernetes both have a powerful query selector syntax, but for the moment awk
will do.
Welcome to the May 2020 report from the Reproducible Builds project. One of the original promises of open source software is that distributed peer review and transparency of process results in enhanced end-user security. Nonetheless, whilst anyone may inspect the source code of free and open source software for malicious flaws, almost all software today is distributed as pre-compiled binaries. This allows nefarious third-parties to compromise systems by injecting malicious code into seemingly secure software during the various compilation and distribution processes. In these reports we outline the most important things that we and the rest of the community have been up to over the past month.
Recent years saw a number of supply chain attacks that leverage the increasing use of open source during software development, which is facilitated by dependency managers that automatically resolve, download and install hundreds of open source packages throughout the software life cycle.In related news, the LineageOS Android distribution announced that a hacker had access to the infrastructure of their servers after exploiting an unpatched vulnerability. Marcin Jachymiak of the Sia decentralised cloud storage platform posted on their blog that their
siac
and siad
utilities can now be built reproducibly:
This means that anyone can recreate the same binaries produced from our official release process. Now anyone can verify that the release binaries were created using the source code we say they were created from. No single person or computer needs to be trusted when producing the binaries now, which greatly reduces the attack surface for Sia users.Synchronicity is a distributed build system for Rust build artifacts which have been published to crates.io. The goal of Synchronicity is to provide a distributed binary transparency system which is independent of any central operator. The Comparison of Linux distributions article on Wikipedia now features a Reproducible Builds column indicating whether distributions approach and progress towards achieving reproducible builds.
binutils
package ships its own, unreproducible, log files in its binary packages. It was followed-up by replies from Chris Lamb and Matthias Klose.
ocaml_cmti_files
toolchain issue.
.apk
packages.
Allan McRae of the ArchLinux project posted their third Reproducible builds progress report to the arch-dev-public
mailing list which includes the following call for help:
We also need help to investigate and fix the packages that fail to reproduce that we have not investigated as of yet.In openSUSE, Bernhard M. Wiedemann published his monthly Reproducible Builds status update.
142
, 143
, 144
, 145
and 146
to Debian, PyPI, etc.
file
now supports recognising JSON data. (#106).changes
and .buildinfo
handling to show all details (including the GnuPG header and footer components) even when referenced files are not present. (#122)BuildinfoFile
comparator (etc.) regardless of whether the associated files (such as the orig.tar.gz
and the .deb
) are present. [ ].buildinfo
, .changes
, etc. [ ]apksigner(1)
. (#121).zip
files. (#116).mobilepovision
files. (#113)differences
typo in the ApkFile
handler. (#127)id="foo"
anchor reference twice in the HTML output, otherwise identically-named parts will not be able to linked to via a #foo
anchor. (#120)#
. [ ]--json
presenter; it will usually be too complicated to be readable by the human anyway. [ ]Command [ ] failed with exit code
messages to remove duplicate exited with exit
but also to note that diffoscope
is interpreting this as an error. [ ]Command [ ] exited with 1
messages. (#126)debian
Python module. [ ]stderr from
if both commands emit the same output. [ ]apksigner
test failures due to lack of binfmt_misc
, eg. on Salsa CI and elsewhere. [ ].travis.yml
as we use Salsa instead. [ ]Dockerfile
improvements:
.dockerignore
file to whitelist files we actually need in our container. (#105)ARG
instead of ENV
when setting up the DEBIAN_FRONTEND
environment variable at runtime. (#103)build-essential
during build so we can install the recommended packages from Git. [ ]shell=False
keyword argument to subprocess.Popen
so that the potentially-unsafe shell=True
is more obvious. [ ]MissingFile
s special handling of deb822
to prevent leaking through abstract layers. [ ][ ]try
/except
block when cleaning up temporary files with respect to the flake8
quality assurance tool. [ ]in_dsc_path
to dsc_in_same_dir
to clarify the use of this variable. [ ]debian_fallback
class [ ] and add descriptions for the file types. [ ]Openssl
command class to OpenSSLPKCS7
to accommodate other command names with this prefix. [ ]--debugger
command-line argument to --pdb
. [ ]stat(2)
birth times (ie. st_birthtime
) in the same way we do with the stat(1)
command s Access:
and Change:
times to fix a nondeterministic build failure in GNU Guix. (#74)LibarchiveMember
s has_same_content
method was called regardless of the underlying type of file. [ ]
debian/py3dist-overrides
to ensure the rpm-python
module is used in package dependencies (#89) and moved to using the new execute_after_*
and execute_before_*
Debhelper rules [ ].
absolute_url
and relative_url
where possible [ ][ ] and move a number of configuration variables to _config.yml
[ ][ ].golang-packaging
(toolchain issue, affecting times in minikube
)jboss-logging-tools
(toolchain issue, affecting date for resteasy
)linux_logo
(sort find
output to avoid inheriting filesystem order)moonjit
(generate reproducible output by default if SOURCE_DATE_EPOCH
is set)vala
(report ASLR nondeterminism)earlyoom
(timestamps in Gzip files)fmt
(Don t install sphinx-build
cached files as they are unneeded & unreproducible)nvidia-settings
(timestamp in Gzip files)ataqv
.elinks
.briquolo
.cryptominisat
.wolfssl
.mistral
.python-watcherclient
.tree-puzzle
.nulib2
.process-cpp
.bowtie2
.properties-cpp
.wand
(forwarded upstream)vows
.libstatgrab
.texi2html
.grub
.systemtap
.mono
.mescc-tools
: Inherit CFLAGS
in a Makefile
, allowing -ffile-prefix-map
/-fdebug-prefix-map
to sanitise build paths (merged upstream).1.8.1-1
to Debian unstable and Bernhard M. Wiedemann fixed an off-by-one error when parsing PNG image modification times. (#16)
In disorderfs, our FUSE-based filesystem that deliberately introduces non-determinism into directory system calls in order to flush out reproducibility issues, Chris Lamb replaced the term dirents in place of directory entries in human-readable output/log messages [ ] and used the astyle source code formatter with the default settings to the main disorderfs.cpp
source file [ ].
Holger Levsen bumped the debhelper-compat level
to 13 in disorderfs [ ] and reprotest [ ], and for the GNU Guix distribution Vagrant Cascadian updated the versions of disorderfs to version 0.5.10 [ ] and diffoscope to version 145 [ ].
libtool
. [ ]_docs
subdirectory to find the _docs/index.md
file after an internal move. (#27)ltmain.sh
etc. in preformatted quotes. [ ]SOURCE_DATE_EPOCH
Python examples onto more lines to prevent visual overflow on the page. [ ]faketime
to the project s Github page. (!57)tests.reproducible-builds.org
that, amongst many other tasks, tracks the status of our reproducibility efforts as well as identifies any regressions that have been introduced. Holger Levsen made the following changes:
let VARIABLE=0
exits with an error. [ ].buildinfo
files with the same name. [ ]/usr
merge variation on Debian unstable. [ ]molly-guard
. [ ]debrebuild
script. [ ][ ][ ][ ].buildinfo
files. [ ][ ]alpine_schroot.sh
script now that a patch for abuild
had been released upstream. [ ]
brcm47xx
target to bcm47xx
. [ ]
jenkins
to run the blacklist
command [ ] and the usual build node maintenance was performed was performed by Holger Levsen [ ][ ][ ], Mattia Rizzolo [ ][ ] and Vagrant Cascadian [ ][ ][ ].
To make the results accessible, storable and create tools around them, they should all follow the same schema, a reproducible builds verification format. The format tries to be as generic as possible to cover all open source projects offering precompiled source code. It stores the rebuilder results of what is reproducible and what not.Hans-Christoph Steiner of the Guardian Project also continued his previous discussion regarding making our website translatable. Lastly, Leo Wandersleb posted a detailed request for feedback on a question of supply chain security and other issues of software review; Leo is the founder of the Wallet Scrutiny project which aims to prove the security of Android Bitcoin Wallets:
Do you own your Bitcoins or do you trust that your app allows you to use your coins while they are actually controlled by them ? Do you have a backup? Do they have a copy they didn t tell you about? Did anybody check the wallet for deliberate backdoors or vulnerabilities? Could anybody check the wallet for those?Elsewhere, Leo had posted instructions on his attempts to reproduce the binaries for the BlueWallet Bitcoin wallet for iOS and Android platforms.
#reproducible-builds
on irc.oftc.net
.
rb-general@lists.reproducible-builds.org
This month s report was written by Bernhard M. Wiedemann, Chris Lamb, Holger Levsen, Jelle van der Waa and Vagrant Cascadian. It was subsequently reviewed by a bunch of Reproducible Builds folks on IRC and the mailing list.
Jline3
(done by @samyak-jn
, myself), and intellij-community-idea
(finished by @The_LoudSpeaker
, Raman Sarda).
kotlin
package residing in m36 s repository had a couple of issues that were needed to be fixed to meet Debian standards, but Kotlin was building fine locally with the mentioned dependencies. :D
patches
as all the changes were made directly to the source, and henceforth fixed rules
and control
files to meet Debian Standards. Debian is very particular about its license policies. The copyright was a pending task that was completed for Good.
The newer package exists at Samyak's repo.
@_hc
) for the help with that. The wiki page for Kotlin exists here.
#debian-mobile
channel on OFTC.
145
. This version includes the following changes:
[ Chris Lamb ]
* Improvements:
- Add support for Apple Xcode mobile provisioning .mobilepovision files.
(Closes: reproducible-builds/diffoscope#113)
- Add support for printing the signatures via apksigner(1).
(Closes: reproducible-builds/diffoscope#121)
- Use SHA256 over MD5 when generating page names for the HTML directory
presenter, validate checksums for files referenced in .changes files
using SHA256 too, and move to using SHA256 in "Too much input for diff"
output too. (Closes: reproducible-builds/diffoscope#124)
- Don't leak the full path of the temporary directory in "Command [..]
exited with 1". (Closes: reproducible-builds/diffoscope#126)
- Identify "iOS App Zip archive data" files as .zip files.
(Closes: reproducible-builds/diffoscope#116)
* Bug fixes:
- Correct "differences" typo in the ApkFile handler.
(Closes: reproducible-builds/diffoscope#127)
* Reporting/output improvements:
- Never emit the same id="foo" TML anchor reference twice, otherwise
identically-named parts will not be able to linked to via "#foo".
(Closes: reproducible-builds/diffoscope#120)
- Never emit HTML with empty "id" anchor lements as it is not possible to
link to "#" (vs "#foo"). We use "#top" as a fallback value so it will
work for the top-level parent container.
- Clarify the message when we cannot find the "debian" Python module.
- Clarify "Command [..] failed with exit code" to remove duplicate "exited
with exit" but also to note that diffoscope is interpreting this as an
error.
- Add descriptions for the 'fallback' Debian module file types.
- Rename the --debugger command-line argument to --pdb.
* Testsuite improvements:
- Prevent CI (and runtime) apksigner test failures due to lack of
binfmt_misc on Salsa CI and elsewhere.
* Codebase improvements:
- Initially add a pair of comments to tidy up a slightly abstraction level
violating code in diffoscope.comparators.mising_file and the
.dsc/.buildinfo file handling, but replace this later by by inlining
MissingFile's special handling of deb822 to prevent leaking through
abstraction layers in the first place.
- Use a BuildinfoFile (etc.) regardless of whether the associated files
such as the orig.tar.gz and the .deb are present, but don't treat them as
actual containers. (Re: reproducible-builds/diffoscope#122)
- Rename the "Openssl" command class to "OpenSSLPKCS7" to accommodate other
commands with this prefix.
- Wrap a docstring across multiple lines, drop an inline pprint import and
comment the HTMLPrintContext class, etc.
[ Emanuel Bronshtein ]
* Avoid build-cache in building the released Docker image.
(Closes: reproducible-builds/diffoscope#123)
[ Holger Levsen ]
* Wrap long lines in older changelog entries.
setxkbmap -layout us -variant altgr-intl
and become a happier programmer./etc/default/keyboard
XKBLAYOUT="us"
XKBVARIANT="altgr-intl"
After restarting the X server, you can check that the settings have been applied withsetxkbmap -print -verbose 10
If using Gnome, you can also set the keyboard layout and variant by changing the schema org.gnome.desktop-inputsources, which will override the desktop-agnostic settings of /etc/default/keyboard.dconf write /org/gnome/desktop/input-sources/sources "[('xkb', 'us+altgr-intl')]"
or navigate with the gui tool dconf-settings to org.gnome.desktop-inputsources and set the value there.# apt install yubikey-manager libu2f-host0
List connected devices on your usb bus:$ lsusb
Bus 002 Device 109: ID 1050:0407 Yubico.com Yubikey 4 OTP+U2F+CCID
Get info about the device capability$ ykman info
Device type: YubiKey 4
Serial number: 1234567
Firmware version: 4.3.7
Enabled USB interfaces: OTP+FIDO+CCID
Applications
OTP Enabled
FIDO U2F Enabled
OpenPGP Enabled
PIV Enabled
OATH Enabled
FIDO2 Not available
The capability which interests us here is FIDO U2F. The Yubikey 4 supports Two Factor Authentification via the U2F standard, and this standard is maintained by the FIDO Industry Association, hence the name. As I plan to only use the FIDO U2F capability of the key, I set FIDO to be the single mode of the key.ykman mode FIDO
Testing web browser interaction with Yubico demo systemNow we need to have to have a browser with support for the U2F standard. Firefox has builtin support since Version 67. Debian 10 Buster has firefox-esr
Version 68, so that will work. For testing yubikeys, the manufacturer has a demo website, where you can test U2F. Go to https://demo.yubico.com and follow the Explore the Yubikey link.Firefox message on the yubikey demo site. A normal site with U2F would not require the extended information, and have a simpler popup message. |
simh
emulator, and tweaked a bit for ARM. asciinema
, also available in the Debian archive, so here is 4.3 BSD, in all its 1986 glory.
binutils
, glibc
and GCC
; Linux
and other useful software like Qemu
) in
2015, but sadly these did not materialise until late 2016 and 2017. One of the
main reasons for the delay was due to the slowness to sort out the copyright
assignment of the code to the FSF (again). Still today, only binutils
and
GCC
are upstreamed, and Linux
and glibc
depend on the Privilege spec being
finished, so it will take a while.
After the experience with OpenRISC and the support in GCC, I didn't want to
invest too much time, lest it all became another dead-end due to lack of
upstreaming so I was just cross-compiling here and there, testing Qemu
(which still today is very limited for this architecture, e.g. no network
support and very limited character and block devices) and trying to find and
report bugs in the implementations, and send patches (although I did not
contribute much in the end).
Incompatible changes in the toolchain
In terms of compiling packages and building-up a repository, things were
complicate, and less mature and stable than the OpenRISC ones were even back in
2014.
In theory, with the Userland spec being frozen, regular programs (below the
Operating System level) compiled at any time could have run today; but in
practice what happened is that there were several rounds of profound or, at
least, disrupting changes in the toolchain before and while being upstreamed,
which made the binary packages that I had built to not work at all (changes in
dynamic loader, registers where arguments are stored when jumping functions,
etc.).
These major breakages happened several times already, and kind of unexpectedly
at least for the people not heavily involved in the implementation.
When the different pieces are upstreamed it is expected that these breakages
won't happen; but still there's at least the fundamental bit of glibc
, which
will probably change things once again in incompatible ways before or while
being upstreamed.
Outside Debian but within the FOSS / Linux world, the main project that I know
of is that some people from Fedora also started a port in mid 2016 and did great
advances, but from what I know they put the project in the freezer in late
2016 until all such problems are resolved they don't want to spend time
rebootstrapping again and again.
What happened recently on the Debian front
In early 2016 I created the page for RISC-V in
the Debian wiki, expecting that things were at last fully stable and the
important bits of the toolchain upstreamed during that year I was too
optimistic.
Some other people (including Debian folks) have been contributing for a while,
in the wiki, mailing lists and IRC channels, and in the RISC-V project mailing
lists you will see their names everywhere.
However, due to the combination of lack of hardware, software not upstreamed and
shortcomings of emulators (chiefly Qemu
) make contributions hard and very
tedious, nothing much happened recently visible to the outside world in terms of
software.
The private repository-in-the-making
In late 2015 and beginning of 2016, having some free time in my hands and
expecting that all things would coalesce quickly, I started to build a
repository of binary packages in a more systematic way, with most of the basic
software that one can expect in a basic Debian system (including things common
to all Linux systems, and also specific Debian software like dpkg
or apt
,
and even aptitude
!).
After that I also built many others outside the basic system (more than 1000
source packages and 2000 or 3000 arch-dependent binary packages in total),
specially popular libraries (e.g. boost
, gtk+
version 2 and 3), interpreters
(several versions of lua
, perl
and python
, also version 2 and 3) and in
general packages that are needed to build many other packages (like doxygen
).
Unfortunately, some of these most interesting packages do not compile cleanly
(more because of obscure or silly errors than proper porting), so they are not
included at the moment.
I intentionally avoided trying to compile thousands of packages in the archive
which would be of nobody's use at this point; but many more could be compiled
without much effort.
About the how, initially I started cross-compiling and using
rebootstrap, which was of
great help in the beginning. Some of the packages that I cross-compiled had
bugs that I did not know how to debug without a live and native (within
emulators) system, so I tried to switch to natively built packages very early
on. For that I needed many packages built natively (like doxygen
or cmake
)
which would be unnecessary if I remained cross-compiling the host tools would
be used in that case.
But this also forced me to eat my own dog food, which even if much slower and
tedious, it was on the whole a more instructive experience; and above all, it
helped to test and make sure that the the tools and the whole stack was working
well enough to build hundreds of packages.
Why the repository-in-the-making was private
Until now I did not attempt to make the repository available on-line, for
several reasons.
First because it would be kind of useless to publish files that were not working
or would soon not work, due to the incompatible changes in the toolchain,
rendering many or most of the packages built useless. And because, for many
months now, I expected that things would stabilise and to have something stable
really soon now but this didn't happen yet.
Second because of lack of resources and time since mid 2016, and because I got
some packages only compiled thanks to (mostly small and unimportant, but
undocumented and unsaved) hacks, often working around temporary bugs and thus
not worth sending upstream; but I couldn't share the binaries without sharing
the full source and fulfill licenses like the GNU
GPL. I did a new round of clean
rebuilds in the last few weeks, just finished, the result is close to 1000
arch-dependent packages.
And third, because of lack of demand. This changed in the last few weeks, when
other people started to ask me to share the results even if incomplete or not
working properly (I had one request in the past, but couldn't oblige in time
at the time).
Finally, the work so far: repository now on-line
So finally, with the great help from Kurt Keville from MIT, and Bytemark
sponsoring a machine where most of the packages were built, here we have the
repository:
The lines for /etc/apt/sources.list
are:
deb [ arch=riscv64 signed-by=/usr/share/keyrings/debian-keyring.gpg ] http://riscv.mit.edu/debian unstable main
deb-src [ signed-by=/usr/share/keyrings/debian-keyring.gpg ] http://riscv.mit.edu/debian unstable main
/usr/share/keyrings/debian-keyring.gpg
, which is part of the package
debian-keyring
(available from Debian and derivatives).
WARNING!!
This repository, though, is very much WIP, incomplete (some package
dependencies cannot be fulfilled, and it's only a small percentage of the Debian
archive, not trying to be comprehensive at the moment) and probably does not
work at all in your system at this point, for the following reasons:
glibc
, gcc
,
linux
, etc.; although I hope that this happens soon after the next stable
release (Stretch) is out of the door and the remaining pieces are upstreamed
(help welcome).glibc
provided it will be difficult that you can get the binaries to
run at all; but there are some packages that are arch-dependent but not too tied
to libc or the dynamic loader will not be affected.
At least you can try one the few static packages present in Debian, like the one
in the package bash-static
. When one removes moving parts like the
dynamic loader and libc, since the basic machine instructions are stable for
several years now, it should work, but I wouldn't discard some dark magic that
prevents even static binaries from working.
Still, I hope that the respository in its current state is useful to some
people, at least for those who requested it. If one has the environment set-up,
it's easy to unpack the contents of the .deb
files and try out the software
(which often is not trivial or very slow to compile, or needs lots of
dependencies to be built first first).
... and finally, even if not useful at all for most people at the moment, by
doing this I also hope that efforts like this spark your interest to contribute
to free software, free hardware, or both! :-)
sng
tool, so I worked on .jpeg and .ico files support.
I initially tried to use exiftool
for extracting metadata, but then I discovered it does not handle .ico files, so I decided
to use a bigger force - ImageMagick s identify
- for this task. I was glad to see it had that handy -format
option I could use
to select only the necessary fields (I found their -verbose
, well, too verbose for the task) and presenting them in the defined
form, negating the need of filtering its output.
What was particulary interesting and important for me in terms of learning: while working on this feature, I discovered that,
at the moment, diffoscope could not handle .ico files at all - img2txt
tool, that was used for retrieving image content, did
not support that type of images. But instead of recognizing this as a bug and resolving it, I started to think of possible
workaround, allowing for retrieving image metadata even after retrieving image content failed.
Definetely not very good thinking. Thanks Mattia Rizzolo for actually recognizing this as a bug and filing it,
and Chris Lamb for fixing it!
.buildinfo
files for Stretch as Debian will not rebuild its source packages and because these binary packages currently in the archive were mostly built with dpkg
> 1.18.11.
reprepro/5.0.0-1 has added support for dealing with .buildinfo
files
that are included in .changes
files. (Closes: #843402)
Reproducible work in other projects
The Chromium project is now working on making their build process (mostly)
deterministic.
Their motivation is to save both "[money] (less hardware is required) and
developer time (reduced latency by having less work to do on the TS and CI)".
Unreproducible bugs filed
arm64
architecture.
arm64
machines with 64GB memory, which allows us to rebuild Debian very fast!Next.